<!DOCTYPE html>
<html  lang="zh">
<head>
    <meta charset="utf-8">
<title>Java 动手写爬虫: 二、 深度爬取 - 一灰灰Blog</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />



    <meta name="description" content="第二篇:深度爬取 前面实现了一个最基础的爬取单网页的爬虫，这一篇则着手解决深度爬取的问题 简单来讲，就是爬了一个网页之后，继续爬这个网页中的链接">
<meta name="keywords" content="爬虫">
<meta property="og:type" content="article">
<meta property="og:title" content="Java 动手写爬虫: 二、 深度爬取">
<meta property="og:url" content="https://blog.hhui.top/hexblog/2017/06/30/Java-动手写爬虫-二、-深度爬取/index.html">
<meta property="og:site_name" content="一灰灰Blog">
<meta property="og:description" content="第二篇:深度爬取 前面实现了一个最基础的爬取单网页的爬虫，这一篇则着手解决深度爬取的问题 简单来讲，就是爬了一个网页之后，继续爬这个网页中的链接">
<meta property="og:locale" content="zh-CN">
<meta property="og:image" content="https://blog.hhui.top/hexblog/images/og_image.png">
<meta property="og:updated_time" content="2018-07-25T14:55:41.000Z">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Java 动手写爬虫: 二、 深度爬取">
<meta name="twitter:description" content="第二篇:深度爬取 前面实现了一个最基础的爬取单网页的爬虫，这一篇则着手解决深度爬取的问题 简单来讲，就是爬了一个网页之后，继续爬这个网页中的链接">
<meta name="twitter:image" content="https://blog.hhui.top/hexblog/images/og_image.png">







<link rel="icon" href="/hexblog/images/avatar.jpg">


<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.7.2/css/bulma.css">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Ubuntu:400,600|Source+Code+Pro">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/highlight.js@9.12.0/styles/docco.css">


    
    
    
    <style>body>.footer,body>.navbar,body>.section{opacity:0}</style>
    

    
    
    
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/lightgallery@1.6.8/dist/css/lightgallery.min.css">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/css/justifiedGallery.min.css">
    

    
    

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/outdatedbrowser@1.1.5/outdatedbrowser/outdatedbrowser.min.css">


    
    
    
    

<link rel="stylesheet" href="/hexblog/css/back-to-top.css">


    
    

    
    
<script>
var _hmt = _hmt || [];
(function() {
    var hm = document.createElement("script");
    hm.src = "//hm.baidu.com/hm.js?028d9e53f991d9739ecc7cc42e13c500";
    var s = document.getElementsByTagName("script")[0];
    s.parentNode.insertBefore(hm, s);
})();
</script>

    
    

    
    
<link rel="stylesheet" href="/hexblog/css/progressbar.css">
<script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>

    
    
    

    
    
    


<link rel="stylesheet" href="/hexblog/css/style.css">
</head>
<body class="is-3-column">
    <nav class="navbar navbar-main">
    <div class="container">
        <div class="navbar-brand is-flex-center">
            <a class="navbar-item navbar-logo" href="/hexblog/">
            
                <img src="/hexblog/images/avatar.jpg" alt="Java 动手写爬虫: 二、 深度爬取" height="28">
            
            </a>
        </div>
        <div class="navbar-menu">
            
            <div class="navbar-start">
                
                <a class="navbar-item"
                href="/hexblog/.">首页</a>
                
                <a class="navbar-item"
                href="/hexblog/archives">归档</a>
                
                <a class="navbar-item"
                href="/hexblog/tags">标签</a>
                
                <a class="navbar-item"
                href="http://spring.hhui.top">Spring</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/Java/">Java</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/Python/">Python</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/DB/">DB</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/Shell/">Shell</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/Quick系列/">Quick系列</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/前端/">前端</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/开源/">开源</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/工具/">工具</a>
                
                <a class="navbar-item"
                href="/hexblog/categories/随笔/">随笔</a>
                
                <a class="navbar-item"
                href="/hexblog/about">关于</a>
                
            </div>
            
            <div class="navbar-end">
                
                    
                    
                    <a class="navbar-item" target="_blank" title="Download on GitHub" href="https://github.com/liuyueyi">
                        
                        <i class="fab fa-github"></i>
                        
                    </a>
                    
                
                
                
                <a class="navbar-item search" title="搜索" href="javascript:;">
                    <i class="fas fa-search"></i>
                </a>
                
            </div>
        </div>
    </div>
</nav>
    
    <section class="section">
        <div class="container">
            <div class="columns">
                <div class="column is-8-tablet is-8-desktop is-7-widescreen has-order-2 column-main"><div class="card">
    
        <span >
            <div class="thumbnail default_logo">
                <br/>
                <span >
                Java 动手写爬虫: 二、 深度爬取
                <br>
                &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                <span style="font-size:0.7em">by 一灰灰</span>
                </span>
            </div>
            </span>
    

    <div class="card-content article ">
        
        <div class="level article-meta is-size-7 is-uppercase is-mobile is-overflow-x-auto">
            <div class="level-left">
                <time class="level-item has-text-grey" datetime="2017-06-30T14:26:16.000Z">2017-06-30</time>
                
                <div class="level-item">
                <a class="has-link-black -link" href="/hexblog/categories/Quick系列/">Quick系列</a>&nbsp;/&nbsp;<a class="has-link-black -link" href="/hexblog/categories/Quick系列/QuickCrawler/">QuickCrawler</a>
                </div>
                
                
                <span class="level-item has-text-grey" style='font-size: 1.2em;'>
                    
                    
                    29 分钟 读完 (大约 4419 个字)
                </span>
                
                
                
                <span class="level-item has-text-grey" id="busuanzi_container_page_pv">
                    <i class="far fa-eye"></i>
                    <span id="busuanzi_value_page_pv">0</span>次访问
                </span>
                
            </div>
        </div>
        
        <h1 class="title is-size-3 is-size-4-mobile has-text-weight-normal">
            
                Java 动手写爬虫: 二、 深度爬取
            
        </h1>
        <div class="content">
            
                <!-- 文章详情页 -->
                <div id="toc" class="toc-article">
                <strong class="toc-title"> 文章目录 </strong>
                    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#第二篇-深度爬取"><span class="toc-text">第二篇:深度爬取</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#I-需求背景"><span class="toc-text">I. 需求背景</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#II-设计"><span class="toc-text">II. 设计</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-基本实现"><span class="toc-text">1. 基本实现</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#说明"><span class="toc-text">说明</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#测试case"><span class="toc-text">测试case</span></a></li></ol></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#III-改进"><span class="toc-text">III. 改进</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#1-问题"><span class="toc-text">1. 问题</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#2-添加链接的过滤"><span class="toc-text">2. 添加链接的过滤</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#3-保存结果"><span class="toc-text">3. 保存结果</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#1-AbstractJob-类中定义一个回调方法"><span class="toc-text">1. AbstractJob 类中定义一个回调方法</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#2-DefaultAbstractCrawlJob-实现爬取网页逻辑的抽象类"><span class="toc-text">2. DefaultAbstractCrawlJob 实现爬取网页逻辑的抽象类</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#3-SimpleCrawlJob"><span class="toc-text">3. SimpleCrawlJob</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#4，使用测试"><span class="toc-text">4，使用测试</span></a></li></ol></li><li class="toc-item toc-level-3"><a class="toc-link" href="#4-爬虫去重"><span class="toc-text">4. 爬虫去重</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#1-IStorage-接口"><span class="toc-text">1. IStorage 接口</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#2-RamStorage-利用Map实现的内存存储"><span class="toc-text">2. RamStorage 利用Map实现的内存存储</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#3-StorageWrapper-封装类"><span class="toc-text">3. StorageWrapper 封装类</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#4-测试"><span class="toc-text">4. 测试</span></a></li></ol></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#IV-小结"><span class="toc-text">IV. 小结</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#相关博文"><span class="toc-text">相关博文</span></a></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#V-其他"><span class="toc-text">V. 其他</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#一灰灰Blog：-https-liuyueyi-github-io-hexblog"><span class="toc-text">一灰灰Blog： https://liuyueyi.github.io/hexblog</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#声明"><span class="toc-text">声明</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#扫描关注"><span class="toc-text">扫描关注</span></a></li></ol></li></ol></li></ol>
                </div>
            

            <h1 id="第二篇-深度爬取"><a href="#第二篇-深度爬取" class="headerlink" title="第二篇:深度爬取"></a>第二篇:深度爬取</h1><blockquote>
<p>前面实现了一个最基础的爬取单网页的爬虫，这一篇则着手解决深度爬取的问题</p>
<p>简单来讲，就是爬了一个网页之后，继续爬这个网页中的链接</p>
</blockquote>
<a id="more"></a>
<h2 id="I-需求背景"><a href="#I-需求背景" class="headerlink" title="I. 需求背景"></a>I. 需求背景</h2><p>背景比较简单和明确，当爬了一个网页之后，目标是不要就此打住，扫描这个网页中的链接，继续爬，所以有几个点需要考虑:</p>
<ul>
<li>哪些链接可以继续爬 ？</li>
<li>是否要一直爬下去，要不要给一个终止符？</li>
<li>新的链接中，提取内容的规则和当前网页的规则不一致可以怎么办？</li>
</ul>
<h2 id="II-设计"><a href="#II-设计" class="headerlink" title="II. 设计"></a>II. 设计</h2><p>针对上面的几点，结合之前的实现结构，在执行 <code>doFetchPage</code> 方法获取网页之后，还得做一些其他的操作</p>
<ul>
<li>扫描网页中的链接，根据过滤规则匹配满足要求的链接</li>
<li>记录一个depth，用于表示爬取的深度，即从最原始的网页出发，到当前页面中间转了几次（讲到这里就有个循环爬取的问题，后面说）</li>
<li>不同的页面提取内容规则不一样，因此可以考虑留一个接口出来，让适用方自己来实现解析网页内容</li>
</ul>
<h3 id="1-基本实现"><a href="#1-基本实现" class="headerlink" title="1. 基本实现"></a>1. 基本实现</h3><blockquote>
<p>开始依然是先把功能点实现，然后再考虑具体的优化细节</p>
</blockquote>
<p>先加一个配置项，表示爬取页面深度; 其次就是保存的结果，得有个容器来暂存， 所以在 <code>SimpleCrawlJob</code> 会新增两个属性</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 批量查询的结果</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-keyword">private</span> List&lt;CrawlResult&gt; crawlResults = <span class="hljs-keyword">new</span> ArrayList&lt;&gt;();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 爬网页的深度, 默认为0， 即只爬取当前网页</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-keyword">private</span> <span class="hljs-keyword">int</span> depth = <span class="hljs-number">0</span>;</span><br></pre></td></tr></table></figure>
<p>因为有深度爬取的过程，所以需要修改一下爬取网页的代码，新增一个 <code>doFetchNetxtPage</code>方法，进行迭代爬取网页，这时，结果匹配处理方法也不能如之前的直接赋值了，稍微改一下即可, 改成返回一个接过实例</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 执行抓取网页</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchPage</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">    doFetchNextPage(<span class="hljs-number">0</span>, <span class="hljs-keyword">this</span>.crawlMeta.getUrl());</span><br><span class="line">    <span class="hljs-keyword">this</span>.crawlResult = <span class="hljs-keyword">this</span>.crawlResults.get(<span class="hljs-number">0</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchNextPage</span><span class="hljs-params">(<span class="hljs-keyword">int</span> currentDepth, String url)</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">    HttpResponse response = HttpUtils.request(<span class="hljs-keyword">new</span> CrawlMeta(url, <span class="hljs-keyword">this</span>.crawlMeta.getSelectorRules()), httpConf);</span><br><span class="line">    String res = EntityUtils.toString(response.getEntity());</span><br><span class="line">    CrawlResult result;</span><br><span class="line">    <span class="hljs-keyword">if</span> (response.getStatusLine().getStatusCode() != <span class="hljs-number">200</span>) &#123; <span class="hljs-comment">// 请求成功</span></span><br><span class="line">        result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">        result.setStatus(response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());</span><br><span class="line">        result.setUrl(crawlMeta.getUrl());</span><br><span class="line">        <span class="hljs-keyword">this</span>.crawlResults.add(result);</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    result = doParse(res);</span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 超过最大深度， 不继续爬</span></span><br><span class="line">    <span class="hljs-keyword">if</span> (currentDepth &gt; depth) &#123;</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line">    <span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">        doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, element.attr(<span class="hljs-string">"href"</span>));</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">private</span> CrawlResult <span class="hljs-title">doParse</span><span class="hljs-params">(String html)</span> </span>&#123;</span><br><span class="line">    Document doc = Jsoup.parse(html);</span><br><span class="line">    </span><br><span class="line">    Map&lt;String, List&lt;String&gt;&gt; map = <span class="hljs-keyword">new</span> HashMap&lt;&gt;(crawlMeta.getSelectorRules().size());</span><br><span class="line">    <span class="hljs-keyword">for</span> (String rule : crawlMeta.getSelectorRules()) &#123;</span><br><span class="line">        List&lt;String&gt; list = <span class="hljs-keyword">new</span> ArrayList&lt;&gt;();</span><br><span class="line">        <span class="hljs-keyword">for</span> (Element element : doc.select(rule)) &#123;</span><br><span class="line">            list.add(element.text());</span><br><span class="line">        &#125;</span><br><span class="line">    </span><br><span class="line">        map.put(rule, list);</span><br><span class="line">    &#125;</span><br><span class="line">    </span><br><span class="line">    </span><br><span class="line">    CrawlResult result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">    result.setHtmlDoc(doc);</span><br><span class="line">    result.setUrl(crawlMeta.getUrl());</span><br><span class="line">    result.setResult(map);</span><br><span class="line">    result.setStatus(CrawlResult.SUCCESS);</span><br><span class="line">    <span class="hljs-keyword">return</span> result;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="说明"><a href="#说明" class="headerlink" title="说明"></a>说明</h4><p>主要的关键代码在 <code>doFetchNextPage</code> 中，这里有两个参数，第一个表示当前url属于爬取的第几层，爬完之后，判断是否超过最大深度，如果没有，则获取出网页中的所有链接，迭代调用一遍</p>
<p>下面主要是获取网页中的跳转链接，直接从jsoup的源码中的example中获取，获取网页中链接的方法</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">// 未超过最大深度， 继续爬网页中的所有链接</span></span><br><span class="line">result = doParse(res);</span><br><span class="line">Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line"><span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">    doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, element.attr(<span class="hljs-string">"href"</span>));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="测试case"><a href="#测试case" class="headerlink" title="测试case"></a>测试case</h4><p>测试代码和之前的差不多，唯一的区别就是指定了爬取的深度，返回结果就不截图了，实在是有点多</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 深度爬</span></span><br><span class="line"><span class="hljs-comment"> * <span class="hljs-doctag">@throws</span> InterruptedException</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-meta">@Test</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testDepthFetch</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>&#123;</span><br><span class="line">    String url = <span class="hljs-string">"https://my.oschina.net/u/566591/blog/1031575"</span>;</span><br><span class="line">    CrawlMeta crawlMeta = <span class="hljs-keyword">new</span> CrawlMeta();</span><br><span class="line">    crawlMeta.setUrl(url);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    SimpleCrawlJob job = <span class="hljs-keyword">new</span> SimpleCrawlJob(<span class="hljs-number">1</span>);</span><br><span class="line">    job.setCrawlMeta(crawlMeta);</span><br><span class="line">    Thread thread = <span class="hljs-keyword">new</span> Thread(job, <span class="hljs-string">"crawlerDepth-test"</span>);</span><br><span class="line">    thread.start();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    thread.join();</span><br><span class="line">    List&lt;CrawlResult&gt; result = job.getCrawlResults();</span><br><span class="line">    System.out.println(result);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h2 id="III-改进"><a href="#III-改进" class="headerlink" title="III. 改进"></a>III. 改进</h2><h3 id="1-问题"><a href="#1-问题" class="headerlink" title="1. 问题"></a>1. 问题</h3><p>上面虽然是实现了目标，但问题却有点多：</p>
<ul>
<li><p>就比如上面的测试case，发现有122个跳转链接，顺序爬速度有点慢</p>
<p><img src="https://static.oschina.net/uploads/img/201706/29235255_R0Hh.png" alt="14987169282459.jpg"></p>
</li>
<li><p>链接中存在重复、页面内锚点、js等各种情况，并不是都满足需求</p>
</li>
<li>最后的结果塞到List中，深度较多时，链接较多时，list可能被撑暴</li>
</ul>
<h3 id="2-添加链接的过滤"><a href="#2-添加链接的过滤" class="headerlink" title="2. 添加链接的过滤"></a>2. 添加链接的过滤</h3><blockquote>
<p>过滤规则，可以划分为两种，正向的匹配，和逆向的排除</p>
</blockquote>
<p>首先是修改配置类 <code>CrawlMeta</code>， 新增两个配置</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 正向的过滤规则</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-meta">@Setter</span></span><br><span class="line"><span class="hljs-meta">@Getter</span></span><br><span class="line"><span class="hljs-keyword">private</span> Set&lt;Pattern&gt; positiveRegex = <span class="hljs-keyword">new</span> HashSet&lt;&gt;();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 逆向的过滤规则</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-meta">@Setter</span></span><br><span class="line"><span class="hljs-meta">@Getter</span></span><br><span class="line"><span class="hljs-keyword">private</span> Set&lt;Pattern&gt; negativeRegex = <span class="hljs-keyword">new</span> HashSet&lt;&gt;();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> Set&lt;Pattern&gt; <span class="hljs-title">addPositiveRegex</span><span class="hljs-params">(String regex)</span> </span>&#123;</span><br><span class="line">    <span class="hljs-keyword">this</span>.positiveRegex.add(Pattern.compile(regex));</span><br><span class="line">    <span class="hljs-keyword">return</span> <span class="hljs-keyword">this</span>.positiveRegex;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> Set&lt;Pattern&gt; <span class="hljs-title">addNegativeRegex</span><span class="hljs-params">(String regex)</span> </span>&#123;</span><br><span class="line">    <span class="hljs-keyword">this</span>.negativeRegex.add(Pattern.compile(regex));</span><br><span class="line">    <span class="hljs-keyword">return</span> <span class="hljs-keyword">this</span>.negativeRegex;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>然后在遍历子链接时，判断一下是否满足需求</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">// doFetchNextPage 方法</span></span><br><span class="line"></span><br><span class="line">Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line">String src;</span><br><span class="line"><span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">    src = element.attr(<span class="hljs-string">"href"</span>);</span><br><span class="line">    <span class="hljs-keyword">if</span> (matchRegex(src)) &#123;</span><br><span class="line">        doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, element.attr(<span class="hljs-string">"href"</span>));</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-comment">// 规则匹配方法</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">matchRegex</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">    Matcher matcher;</span><br><span class="line">    <span class="hljs-keyword">for</span>(Pattern pattern: crawlMeta.getPositiveRegex()) &#123;</span><br><span class="line">        matcher = pattern.matcher(url);</span><br><span class="line">        <span class="hljs-keyword">if</span> (matcher.find()) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">for</span>(Pattern pattern: crawlMeta.getNegativeRegex()) &#123;</span><br><span class="line">        matcher = pattern.matcher(url);</span><br><span class="line">        <span class="hljs-keyword">if</span>(matcher.find()) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">return</span> crawlMeta.getPositiveRegex().size() == <span class="hljs-number">0</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>上面主要是通过正则来进行过滤，暂不考虑正则带来的开销问题，至少是解决了一个过滤的问题</p>
<p>但是，但是，如果网页中的链接是相对路径的话，会怎么样</p>
<p>直接使用 Jsoup来测试一个网页，看获取的link地址为什么</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">// 获取网页中的所有链接</span></span><br><span class="line"><span class="hljs-meta">@Test</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testGetLink</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> IOException </span>&#123;</span><br><span class="line">    String url = <span class="hljs-string">"http://chengyu.911cha.com/zishu_3_p1.html"</span>;</span><br><span class="line"></span><br><span class="line">    Connection httpConnection = HttpConnection.connect(url)</span><br><span class="line">            .header(<span class="hljs-string">"accept"</span>, <span class="hljs-string">"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"</span>)</span><br><span class="line">            .header(<span class="hljs-string">"connection"</span>, <span class="hljs-string">"Keep-Alive"</span>)</span><br><span class="line">            .header(<span class="hljs-string">"user-agent"</span>, <span class="hljs-string">"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"</span>);</span><br><span class="line"></span><br><span class="line">    Document doc = httpConnection.get();</span><br><span class="line">    Elements links = doc.select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line"></span><br><span class="line">    print(<span class="hljs-string">"\nLinks: (%d)"</span>, links.size());</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>看下取出的链接</p>
<p><img src="https://static.oschina.net/uploads/img/201706/29235351_2GVq.png" alt="1.png"></p>
<p>根据上面的测试，获取的链接如果是相对地址，则会有问题，需要有一个转化的过程，这个改动比较简单，jsoup本身是支持的</p>
<p>改一行即可</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">// 解析为documnet对象时，指定 baseUrl</span></span><br><span class="line"><span class="hljs-comment">// 上面的代码结构会做一点修改，后面会说到</span></span><br><span class="line">Document doc = Jsoup.parse(html, url);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="hljs-comment">// 获取链接时，前面添加abs</span></span><br><span class="line">src = element.attr(<span class="hljs-string">"abs:href"</span>);</span><br></pre></td></tr></table></figure>
<p><img src="https://static.oschina.net/uploads/img/201706/29235500_6lxk.png" alt="3.png"></p>
<h3 id="3-保存结果"><a href="#3-保存结果" class="headerlink" title="3. 保存结果"></a>3. 保存结果</h3><p>当爬取的数据量较多时，将结果都保存在内存中，并不是一个好的选择，假色每个网页中，满足规则的是有10个，那么depth=n， 则从第一个网页出发，最终会得到</p>
<p><code>1 + 10 + ... + 10^n = (10^(n+1) - 1) / 9</code></p>
<p>显然在实际情况中是不可取的，因此可以改造一下，获取数据后给一个回调，让用户自己来选择如何处理结果，这时 <code>SimpleCrawelJob</code> 的结构基本上满足不了需求了</p>
<p>重新开始设计</p>
<h4 id="1-AbstractJob-类中定义一个回调方法"><a href="#1-AbstractJob-类中定义一个回调方法" class="headerlink" title="1. AbstractJob 类中定义一个回调方法"></a>1. <code>AbstractJob</code> 类中定义一个回调方法</h4><figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 解析完网页后的回调方法</span></span><br><span class="line"><span class="hljs-comment"> *</span></span><br><span class="line"><span class="hljs-comment"> * <span class="hljs-doctag">@param</span> crawlResult</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">abstract</span> <span class="hljs-keyword">void</span> <span class="hljs-title">visit</span><span class="hljs-params">(CrawlResult crawlResult)</span></span>;</span><br></pre></td></tr></table></figure>
<h4 id="2-DefaultAbstractCrawlJob-实现爬取网页逻辑的抽象类"><a href="#2-DefaultAbstractCrawlJob-实现爬取网页逻辑的抽象类" class="headerlink" title="2. DefaultAbstractCrawlJob 实现爬取网页逻辑的抽象类"></a>2. <code>DefaultAbstractCrawlJob</code> 实现爬取网页逻辑的抽象类</h4><p>这个类实现爬取网页的主要逻辑，也就是将之前的<code>SimpleCrwalJob</code>的实现拷贝过来，区别是干掉了返回结果; 顺带修了一个小bug 😢</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * Created by yihui on 2017/6/29.</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-meta">@Getter</span></span><br><span class="line"><span class="hljs-meta">@Setter</span></span><br><span class="line"><span class="hljs-meta">@NoArgsConstructor</span></span><br><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-keyword">abstract</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">DefaultAbstractCrawlJob</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">AbstractJob</span> </span>&#123;</span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 配置项信息</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-keyword">private</span> CrawlMeta crawlMeta;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * http配置信息</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-keyword">private</span> CrawlHttpConf httpConf = <span class="hljs-keyword">new</span> CrawlHttpConf();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 爬网页的深度, 默认为0， 即只爬取当前网页</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-keyword">protected</span> <span class="hljs-keyword">int</span> depth = <span class="hljs-number">0</span>;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-title">DefaultAbstractCrawlJob</span><span class="hljs-params">(<span class="hljs-keyword">int</span> depth)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">this</span>.depth = depth;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 执行抓取网页</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchPage</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">        doFetchNextPage(<span class="hljs-number">0</span>, <span class="hljs-keyword">this</span>.crawlMeta.getUrl());</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchNextPage</span><span class="hljs-params">(<span class="hljs-keyword">int</span> currentDepth, String url)</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">        CrawlMeta subMeta = <span class="hljs-keyword">new</span> CrawlMeta(url, <span class="hljs-keyword">this</span>.crawlMeta.getSelectorRules(), <span class="hljs-keyword">this</span>.crawlMeta.getPositiveRegex(), <span class="hljs-keyword">this</span>.crawlMeta.getNegativeRegex());</span><br><span class="line">        HttpResponse response = HttpUtils.request(subMeta, httpConf);</span><br><span class="line">        String res = EntityUtils.toString(response.getEntity());</span><br><span class="line">        CrawlResult result;</span><br><span class="line">        <span class="hljs-keyword">if</span> (response.getStatusLine().getStatusCode() != <span class="hljs-number">200</span>) &#123; <span class="hljs-comment">// 请求成功</span></span><br><span class="line">            result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">            result.setStatus(response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());</span><br><span class="line">            result.setUrl(crawlMeta.getUrl());</span><br><span class="line">            <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line">            <span class="hljs-keyword">return</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        <span class="hljs-comment">// 网页解析</span></span><br><span class="line">        result = doParse(res, subMeta);</span><br><span class="line">        <span class="hljs-comment">// 回调用户的网页内容解析方法</span></span><br><span class="line">        <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        <span class="hljs-comment">// 超过最大深度， 不继续爬</span></span><br><span class="line">        <span class="hljs-keyword">if</span> (currentDepth &gt; depth) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line">        String src;</span><br><span class="line">        <span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">            <span class="hljs-comment">// 确保将相对地址转为绝对地址</span></span><br><span class="line">            src = element.attr(<span class="hljs-string">"abs:href"</span>);</span><br><span class="line">            <span class="hljs-keyword">if</span> (matchRegex(src)) &#123;</span><br><span class="line">                doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, src);</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> CrawlResult <span class="hljs-title">doParse</span><span class="hljs-params">(String html, CrawlMeta meta)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-comment">// 指定baseUrl， 否则利用 abs:href 获取链接会出错</span></span><br><span class="line">        Document doc = Jsoup.parse(html, meta.getUrl());</span><br><span class="line"></span><br><span class="line">        Map&lt;String, List&lt;String&gt;&gt; map = <span class="hljs-keyword">new</span> HashMap&lt;&gt;(meta.getSelectorRules().size());</span><br><span class="line">        <span class="hljs-keyword">for</span> (String rule : crawlMeta.getSelectorRules()) &#123;</span><br><span class="line">            List&lt;String&gt; list = <span class="hljs-keyword">new</span> ArrayList&lt;&gt;();</span><br><span class="line">            <span class="hljs-keyword">for</span> (Element element : doc.select(rule)) &#123;</span><br><span class="line">                list.add(element.text());</span><br><span class="line">            &#125;</span><br><span class="line"></span><br><span class="line">            map.put(rule, list);</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        CrawlResult result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">        result.setHtmlDoc(doc);</span><br><span class="line">        result.setUrl(meta.getUrl());</span><br><span class="line">        result.setResult(map);</span><br><span class="line">        result.setStatus(CrawlResult.SUCCESS);</span><br><span class="line">        <span class="hljs-keyword">return</span> result;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">matchRegex</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        Matcher matcher;</span><br><span class="line">        <span class="hljs-keyword">for</span>(Pattern pattern: crawlMeta.getPositiveRegex()) &#123;</span><br><span class="line">            matcher = pattern.matcher(url);</span><br><span class="line">            <span class="hljs-keyword">if</span> (matcher.find()) &#123;</span><br><span class="line">                <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        <span class="hljs-keyword">for</span>(Pattern pattern: crawlMeta.getNegativeRegex()) &#123;</span><br><span class="line">            matcher = pattern.matcher(url);</span><br><span class="line">            <span class="hljs-keyword">if</span>(matcher.find()) &#123;</span><br><span class="line">                <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        <span class="hljs-keyword">return</span> crawlMeta.getPositiveRegex().size() == <span class="hljs-number">0</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="3-SimpleCrawlJob"><a href="#3-SimpleCrawlJob" class="headerlink" title="3. SimpleCrawlJob"></a>3. <code>SimpleCrawlJob</code></h4><p>重写这个简单爬虫任务的实现，因为主要逻辑在 <code>DefaultAbstractCrawlJob</code>中已经实现了，所以直接继承过来即可</p>
<p>主要关注的就是 <code>visit</code> 方法，这里就是爬取网页之后的回调，这个最简单的爬虫任务，就是将结果保存在内存中</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * 最简单的一个爬虫任务</span></span><br><span class="line"><span class="hljs-comment"> * &lt;p&gt;</span></span><br><span class="line"><span class="hljs-comment"> * Created by yihui on 2017/6/27.</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-meta">@Getter</span></span><br><span class="line"><span class="hljs-meta">@Setter</span></span><br><span class="line"><span class="hljs-meta">@NoArgsConstructor</span></span><br><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">SimpleCrawlJob</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">DefaultAbstractCrawlJob</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 存储爬取的结果</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-keyword">private</span> CrawlResult crawlResult;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 批量查询的结果</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-keyword">private</span> List&lt;CrawlResult&gt; crawlResults = <span class="hljs-keyword">new</span> ArrayList&lt;&gt;();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-title">SimpleCrawlJob</span><span class="hljs-params">(<span class="hljs-keyword">int</span> depth)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">super</span>(depth);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-meta">@Override</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">void</span> <span class="hljs-title">visit</span><span class="hljs-params">(CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">        crawlResults.add(crawlResult);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> CrawlResult <span class="hljs-title">getCrawlResult</span><span class="hljs-params">()</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">if</span>(crawlResults.size() == <span class="hljs-number">0</span>) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="hljs-keyword">return</span> crawlResults.get(<span class="hljs-number">0</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="4，使用测试"><a href="#4，使用测试" class="headerlink" title="4，使用测试"></a>4，使用测试</h4><p>和之前没有任何区别，先来个简单的</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-meta">@Test</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testDepthFetch</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>&#123;</span><br><span class="line">    String url = <span class="hljs-string">"http://chengyu.911cha.com/zishu_3_p1.html"</span>;</span><br><span class="line">    CrawlMeta crawlMeta = <span class="hljs-keyword">new</span> CrawlMeta();</span><br><span class="line">    crawlMeta.setUrl(url);</span><br><span class="line">    crawlMeta.addPositiveRegex(<span class="hljs-string">"http://chengyu.911cha.com/zishu_3_p([0-9]+).html"</span>);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    SimpleCrawlJob job = <span class="hljs-keyword">new</span> SimpleCrawlJob(<span class="hljs-number">1</span>);</span><br><span class="line">    job.setCrawlMeta(crawlMeta);</span><br><span class="line">    Thread thread = <span class="hljs-keyword">new</span> Thread(job, <span class="hljs-string">"crawlerDepth-test"</span>);</span><br><span class="line">    thread.start();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    thread.join();</span><br><span class="line">    List&lt;CrawlResult&gt; result = job.getCrawlResults();</span><br><span class="line">    System.out.println(result);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>运行截图</p>
<p><img src="https://static.oschina.net/uploads/img/201706/29235548_KrBA.png" alt="1.png"></p>
<p>直接使用 <code>DefaultAbstractCrawl</code> 抽象类的回调来进行测试</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-meta">@Test</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testSelfCwralFetch</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>&#123;</span><br><span class="line">    String url = <span class="hljs-string">"http://chengyu.911cha.com/zishu_3_p1.html"</span>;</span><br><span class="line">    CrawlMeta crawlMeta = <span class="hljs-keyword">new</span> CrawlMeta();</span><br><span class="line">    crawlMeta.setUrl(url);</span><br><span class="line">    crawlMeta.addPositiveRegex(<span class="hljs-string">"http://chengyu.911cha.com/zishu_3_p([0-9]+).html"</span>);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    DefaultAbstractCrawlJob job = <span class="hljs-keyword">new</span> DefaultAbstractCrawlJob(<span class="hljs-number">1</span>) &#123;</span><br><span class="line">        <span class="hljs-meta">@Override</span></span><br><span class="line">        <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">void</span> <span class="hljs-title">visit</span><span class="hljs-params">(CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">            System.out.println(crawlResult.getUrl());</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;;</span><br><span class="line">    job.setCrawlMeta(crawlMeta);</span><br><span class="line">    Thread thread = <span class="hljs-keyword">new</span> Thread(job, <span class="hljs-string">"crawlerDepth-test"</span>);</span><br><span class="line">    thread.start();</span><br><span class="line"></span><br><span class="line">    thread.join();</span><br><span class="line">    System.out.println(<span class="hljs-string">"over"</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h3 id="4-爬虫去重"><a href="#4-爬虫去重" class="headerlink" title="4. 爬虫去重"></a>4. 爬虫去重</h3><blockquote>
<p>从上面可以发现，重复爬取是比较浪费的事情，因此去重是非常有必要的；一般想法是将爬过的url都标记一下，每次爬之前判断是否已经爬过了</p>
</blockquote>
<p>依然先是采用最low的方法，搞一个Set来记录所有爬取的url，因为具体的爬虫任务设计的是多线程的，所以这个Set是要求多线程共享的</p>
<p>此外考虑到去重的手段比较多，我们目前虽然只是采用的内存中加一个缓存表，但不妨碍我们设计的时候，采用面向接口的方式</p>
<h4 id="1-IStorage-接口"><a href="#1-IStorage-接口" class="headerlink" title="1. IStorage 接口"></a>1. <code>IStorage</code> 接口</h4><blockquote>
<p>提供存记录，判断记录是否存在的方法</p>
</blockquote>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">interface</span> <span class="hljs-title">IStorage</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 若爬取的URL不在storage中， 则写入； 否则忽略</span></span><br><span class="line"><span class="hljs-comment">     *</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url 爬取的网址</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@return</span> true 表示写入成功， 即之前没有这条记录； false 则表示之前已经有记录了</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">boolean</span> <span class="hljs-title">putIfNotExist</span><span class="hljs-params">(String url, CrawlResult result)</span></span>;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 判断是否存在</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@return</span></span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">boolean</span> <span class="hljs-title">contains</span><span class="hljs-params">(String url)</span></span>;</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="2-RamStorage-利用Map实现的内存存储"><a href="#2-RamStorage-利用Map实现的内存存储" class="headerlink" title="2. RamStorage 利用Map实现的内存存储"></a>2. <code>RamStorage</code> 利用Map实现的内存存储</h4><figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">RamStorage</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">IStorage</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> Map&lt;String, CrawlResult&gt; map = <span class="hljs-keyword">new</span> ConcurrentHashMap&lt;&gt;();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-meta">@Override</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">putIfNotExist</span><span class="hljs-params">(String url, CrawlResult result)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">if</span>(map.containsKey(url)) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        map.put(url, result);</span><br><span class="line">        <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-meta">@Override</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">contains</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">return</span> map.containsKey(url);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="3-StorageWrapper-封装类"><a href="#3-StorageWrapper-封装类" class="headerlink" title="3. StorageWrapper 封装类"></a>3. <code>StorageWrapper</code> 封装类</h4><blockquote>
<p>这个封装类要求多线程共享，所以我们采用单例模式，保证只有一个实例</p>
</blockquote>
<p>一个最原始的实现方式如下(暂不考虑其中比较猥琐的storage实例化方式)</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">StorageWrapper</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> StorageWrapper instance = <span class="hljs-keyword">new</span> StorageWrapper();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> IStorage storage;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> StorageWrapper <span class="hljs-title">getInstance</span><span class="hljs-params">()</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">return</span> instance;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-title">StorageWrapper</span><span class="hljs-params">()</span> </span>&#123;</span><br><span class="line">        storage = <span class="hljs-keyword">new</span> RamStorage();</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 判断url是否被爬取过</span></span><br><span class="line"><span class="hljs-comment">     *</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@return</span></span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">ifUrlFetched</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">return</span> storage.contains(url);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 爬完之后， 新增一条爬取记录</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> crawlResult</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">addFetchRecord</span><span class="hljs-params">(String url, CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">        storage.putIfNotExist(url, crawlResult);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>这样一个简单的保存爬取历史记录的容器就有了，那么在爬取时，就需要事前判断一下</p>
<p>对应的 <code>DefaultAbstractCrawlJob#doFetchNextPage</code> 方法更新如下</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">// fixme 非线程安全</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchNextPage</span><span class="hljs-params">(<span class="hljs-keyword">int</span> currentDepth, String url)</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">    <span class="hljs-keyword">if</span> (StorageWrapper.getInstance().ifUrlFetched(url)) &#123;</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    CrawlMeta subMeta = <span class="hljs-keyword">new</span> CrawlMeta(url, <span class="hljs-keyword">this</span>.crawlMeta.getSelectorRules(), <span class="hljs-keyword">this</span>.crawlMeta.getPositiveRegex(), <span class="hljs-keyword">this</span>.crawlMeta.getNegativeRegex());</span><br><span class="line">    HttpResponse response = HttpUtils.request(subMeta, httpConf);</span><br><span class="line">    String res = EntityUtils.toString(response.getEntity());</span><br><span class="line">    CrawlResult result;</span><br><span class="line">    <span class="hljs-keyword">if</span> (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) &#123; <span class="hljs-comment">// 请求成功</span></span><br><span class="line">        result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">        result.setStatus(response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());</span><br><span class="line">        result.setUrl(crawlMeta.getUrl());</span><br><span class="line">        <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 网页解析</span></span><br><span class="line">    result = doParse(res, subMeta);</span><br><span class="line">    StorageWrapper.getInstance().addFetchRecord(url, result);</span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 回调用户的网页内容解析方法</span></span><br><span class="line">    <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 超过最大深度， 不继续爬</span></span><br><span class="line">    <span class="hljs-keyword">if</span> (currentDepth &gt; depth) &#123;</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line">    String src;</span><br><span class="line">    <span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">        <span class="hljs-comment">// 确保将相对地址转为绝对地址</span></span><br><span class="line">        src = element.attr(<span class="hljs-string">"abs:href"</span>);</span><br><span class="line">        <span class="hljs-keyword">if</span> (matchRegex(src)) &#123;</span><br><span class="line">            doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, src);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>如果仔细看上面的方法，就会发现在多线程环境下，依然可能存在重复爬取的情况</p>
<p>如有两个<code>CrawlJob</code>任务，若爬取的是同一个url，第一个任务爬取完，还没有回写到<code>Storage</code>时，第二个任务开始爬，这时，事前判断没有记录，然后通过之后开始爬，这时就依然会出现重复爬的问题</p>
<p>要解决这个问题，一个简单的方法就是加锁，在判断一个url没有被爬时，到回写一条爬取结果这段期间，加一个保护锁</p>
<p><code>StorageWrapper</code> 更新后如下</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment"> * Created by yihui on 2017/6/29.</span></span><br><span class="line"><span class="hljs-comment"> */</span></span><br><span class="line"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">StorageWrapper</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> StorageWrapper instance = <span class="hljs-keyword">new</span> StorageWrapper();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> IStorage storage;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-keyword">private</span> Map&lt;String, Lock&gt; lockMap = <span class="hljs-keyword">new</span> ConcurrentHashMap&lt;&gt;();</span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> StorageWrapper <span class="hljs-title">getInstance</span><span class="hljs-params">()</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">return</span> instance;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-title">StorageWrapper</span><span class="hljs-params">()</span> </span>&#123;</span><br><span class="line">        storage = <span class="hljs-keyword">new</span> RamStorage();</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 判断url是否被爬取过; 是则返回true； 否这返回false， 并上锁</span></span><br><span class="line"><span class="hljs-comment">     *</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@return</span></span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">ifUrlFetched</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">if</span>(storage.contains(url)) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="hljs-keyword">synchronized</span> (<span class="hljs-keyword">this</span>) &#123;</span><br><span class="line">            <span class="hljs-keyword">if</span> (!lockMap.containsKey(url)) &#123;</span><br><span class="line">                <span class="hljs-comment">// 不存在时，加一个锁</span></span><br><span class="line">                lockMap.put(url, <span class="hljs-keyword">new</span> ReentrantLock());</span><br><span class="line">            &#125;</span><br><span class="line"></span><br><span class="line">            <span class="hljs-keyword">this</span>.lock(url);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">            <span class="hljs-keyword">if</span> (storage.contains(url)) &#123;</span><br><span class="line">                <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;</span><br><span class="line">            &#125;</span><br><span class="line"><span class="hljs-comment">//            System.out.println(Thread.currentThread() + " lock url: " + url);</span></span><br><span class="line">            <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">/**</span></span><br><span class="line"><span class="hljs-comment">     * 爬完之后， 新增一条爬取记录</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> url</span></span><br><span class="line"><span class="hljs-comment">     * <span class="hljs-doctag">@param</span> crawlResult</span></span><br><span class="line"><span class="hljs-comment">     */</span></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">addFetchRecord</span><span class="hljs-params">(String url, CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">        <span class="hljs-keyword">try</span> &#123;</span><br><span class="line">            <span class="hljs-keyword">if</span> (crawlResult != <span class="hljs-keyword">null</span>) &#123;</span><br><span class="line">                storage.putIfNotExist(url, crawlResult);</span><br><span class="line">                <span class="hljs-keyword">this</span>.unlock(url);</span><br><span class="line">            &#125;</span><br><span class="line">        &#125; <span class="hljs-keyword">catch</span> (Exception e) &#123;</span><br><span class="line">            System.out.println(Thread.currentThread().getName() + <span class="hljs-string">" result: "</span> + url + <span class="hljs-string">" e: "</span> + e);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">lock</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        lockMap.get(url).lock();</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">unlock</span><span class="hljs-params">(String url)</span> </span>&#123;</span><br><span class="line">        lockMap.get(url).unlock();</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>使用处，稍稍变动如下</p>
<figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-keyword">void</span> <span class="hljs-title">doFetchNextPage</span><span class="hljs-params">(<span class="hljs-keyword">int</span> currentDepth, String url)</span> <span class="hljs-keyword">throws</span> Exception </span>&#123;</span><br><span class="line">    CrawlResult result = <span class="hljs-keyword">null</span>;</span><br><span class="line">    <span class="hljs-keyword">try</span> &#123;</span><br><span class="line">        <span class="hljs-comment">// 判断是否爬过；未爬取，则上锁并继续爬取网页</span></span><br><span class="line">        <span class="hljs-keyword">if</span> (StorageWrapper.getInstance().ifUrlFetched(url)) &#123;</span><br><span class="line">            <span class="hljs-keyword">return</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        CrawlMeta subMeta = <span class="hljs-keyword">new</span> CrawlMeta(url, <span class="hljs-keyword">this</span>.crawlMeta.getSelectorRules(), <span class="hljs-keyword">this</span>.crawlMeta.getPositiveRegex(), <span class="hljs-keyword">this</span>.crawlMeta.getNegativeRegex());</span><br><span class="line">        HttpResponse response = HttpUtils.request(subMeta, httpConf);</span><br><span class="line">        String res = EntityUtils.toString(response.getEntity());</span><br><span class="line">        <span class="hljs-keyword">if</span> (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) &#123; <span class="hljs-comment">// 请求成功</span></span><br><span class="line">            result = <span class="hljs-keyword">new</span> CrawlResult();</span><br><span class="line">            result.setStatus(response.getStatusLine().getStatusCode(), response.getStatusLine().getReasonPhrase());</span><br><span class="line">            result.setUrl(crawlMeta.getUrl());</span><br><span class="line">            <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line">            <span class="hljs-keyword">return</span>;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">        <span class="hljs-comment">// 网页解析</span></span><br><span class="line">        result = doParse(res, subMeta);</span><br><span class="line">    &#125; <span class="hljs-keyword">finally</span> &#123;</span><br><span class="line">        <span class="hljs-comment">// 添加一条记录， 并释放锁</span></span><br><span class="line">        StorageWrapper.getInstance().addFetchRecord(url, result);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 回调用户的网页内容解析方法</span></span><br><span class="line">    <span class="hljs-keyword">this</span>.visit(result);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    <span class="hljs-comment">// 超过最大深度， 不继续爬</span></span><br><span class="line">    <span class="hljs-keyword">if</span> (currentDepth &gt; depth) &#123;</span><br><span class="line">        <span class="hljs-keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    Elements elements = result.getHtmlDoc().select(<span class="hljs-string">"a[href]"</span>);</span><br><span class="line">    String src;</span><br><span class="line">    <span class="hljs-keyword">for</span>(Element element: elements) &#123;</span><br><span class="line">        <span class="hljs-comment">// 确保将相对地址转为绝对地址</span></span><br><span class="line">        src = element.attr(<span class="hljs-string">"abs:href"</span>);</span><br><span class="line">        <span class="hljs-keyword">if</span> (matchRegex(src)) &#123;</span><br><span class="line">            doFetchNextPage(currentDepth + <span class="hljs-number">1</span>, src);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h4 id="4-测试"><a href="#4-测试" class="headerlink" title="4. 测试"></a>4. 测试</h4><figure class="highlight java hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"><span class="hljs-meta">@Test</span></span><br><span class="line"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testSelfCwralFetch</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>&#123;</span><br><span class="line">    String url = <span class="hljs-string">"http://chengyu.t086.com/gushi/1.htm"</span>;</span><br><span class="line">    CrawlMeta crawlMeta = <span class="hljs-keyword">new</span> CrawlMeta();</span><br><span class="line">    crawlMeta.setUrl(url);</span><br><span class="line">    crawlMeta.addPositiveRegex(<span class="hljs-string">"http://chengyu.t086.com/gushi/[0-9]+\\.htm$"</span>);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    DefaultAbstractCrawlJob job = <span class="hljs-keyword">new</span> DefaultAbstractCrawlJob(<span class="hljs-number">1</span>) &#123;</span><br><span class="line">        <span class="hljs-meta">@Override</span></span><br><span class="line">        <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">void</span> <span class="hljs-title">visit</span><span class="hljs-params">(CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">            System.out.println(<span class="hljs-string">"job1 &gt;&gt;&gt; "</span> + crawlResult.getUrl());</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;;</span><br><span class="line">    job.setCrawlMeta(crawlMeta);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    String url2 = <span class="hljs-string">"http://chengyu.t086.com/gushi/2.htm"</span>;</span><br><span class="line">    CrawlMeta crawlMeta2 = <span class="hljs-keyword">new</span> CrawlMeta();</span><br><span class="line">    crawlMeta2.setUrl(url2);</span><br><span class="line">    crawlMeta2.addPositiveRegex(<span class="hljs-string">"http://chengyu.t086.com/gushi/[0-9]+\\.htm$"</span>);</span><br><span class="line">    DefaultAbstractCrawlJob job2 = <span class="hljs-keyword">new</span> DefaultAbstractCrawlJob(<span class="hljs-number">1</span>) &#123;</span><br><span class="line">        <span class="hljs-meta">@Override</span></span><br><span class="line">        <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">void</span> <span class="hljs-title">visit</span><span class="hljs-params">(CrawlResult crawlResult)</span> </span>&#123;</span><br><span class="line">            System.out.println(<span class="hljs-string">"job2 &gt;&gt;&gt; "</span> + crawlResult.getUrl());</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;;</span><br><span class="line">    job2.setCrawlMeta(crawlMeta2);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    Thread thread = <span class="hljs-keyword">new</span> Thread(job, <span class="hljs-string">"crawlerDepth-test"</span>);</span><br><span class="line">    Thread thread2 = <span class="hljs-keyword">new</span> Thread(job2, <span class="hljs-string">"crawlerDepth-test2"</span>);</span><br><span class="line">    thread.start();</span><br><span class="line">    thread2.start();</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">    thread.join();</span><br><span class="line">    thread2.join();</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>输出如下</p>
<figure class="highlight plain hljs"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">job2 &gt;&gt;&gt; http://chengyu.t086.com/gushi/2.htm</span><br><span class="line">job1 &gt;&gt;&gt; http://chengyu.t086.com/gushi/1.htm</span><br><span class="line">job1 &gt;&gt;&gt; http://chengyu.t086.com/gushi/3.htm</span><br><span class="line">job2 &gt;&gt;&gt; http://chengyu.t086.com/gushi/4.htm</span><br><span class="line">job1 &gt;&gt;&gt; http://chengyu.t086.com/gushi/5.htm</span><br><span class="line">job1 &gt;&gt;&gt; http://chengyu.t086.com/gushi/6.htm</span><br><span class="line">job1 &gt;&gt;&gt; http://chengyu.t086.com/gushi/7.htm</span><br></pre></td></tr></table></figure>
<hr>
<h2 id="IV-小结"><a href="#IV-小结" class="headerlink" title="IV. 小结"></a>IV. 小结</h2><p>这一篇的博文有点多，到这里其实上面一些提出的问题还没有解决，留待下一篇博文来fix掉, 下面则主要说明下本篇的要点</p>
<ol>
<li><p>深度爬取</p>
<p>这里使用了迭代的思路，爬到一个网页之后，判断是否需要停止，不停止，则把该网页中的链接捞出来，继续爬；关键点</p>
<ul>
<li>利用 Jsoup 获取网页中所有链接（注意相对路径转绝对路径的用法）</li>
<li>循环迭代</li>
</ul>
</li>
<li><p>过滤</p>
<p>过滤，主要利用正则来匹配链接；这里需要注意一下几点</p>
<ul>
<li>正向过滤</li>
<li>负向过滤</li>
</ul>
</li>
<li><p>去重</p>
<p>如何保证一个链接被爬了之后，不会被重复进行爬取？</p>
<ul>
<li>记录爬取历史</li>
<li>注意多线程安全问题</li>
<li>加锁（一把锁会导致性能降低，这里采用了一个url对应一个锁，注意看实现细节，较多的坑）</li>
</ul>
</li>
</ol>
<p><strong>遗留问题</strong></p>
<ol>
<li>失败重试</li>
<li>爬网页中链接不应该串行进行</li>
<li>频率控制（太快可能会被反扒干掉）</li>
</ol>
<p>源码地址： <a href="https://github.com/liuyueyi/quick-crawler/releases/tag/v0.003" target="_blank" rel="noopener">https://github.com/liuyueyi/quick-crawler/releases/tag/v0.003</a></p>
<p>对应tag ：<a href="https://github.com/liuyueyi/quick-crawler/releases/tag/v0.003" target="_blank" rel="noopener">v0.003</a></p>
<h3 id="相关博文"><a href="#相关博文" class="headerlink" title="相关博文"></a>相关博文</h3><p><a href="https://liuyueyi.github.io/hexblog/categories/%E6%8A%80%E6%9C%AF/Quick%E7%B3%BB%E5%88%97%E9%A1%B9%E7%9B%AE/QuickCrawler/" target="_blank" rel="noopener">Quick-Crawel爬虫系列博文</a></p>
<h2 id="V-其他"><a href="#V-其他" class="headerlink" title="V. 其他"></a>V. 其他</h2><h3 id="一灰灰Blog：-https-liuyueyi-github-io-hexblog"><a href="#一灰灰Blog：-https-liuyueyi-github-io-hexblog" class="headerlink" title="一灰灰Blog： https://liuyueyi.github.io/hexblog"></a><a href="https://liuyueyi.github.io/hexblog" target="_blank" rel="noopener">一灰灰Blog</a>： <a href="https://liuyueyi.github.io/hexblog" target="_blank" rel="noopener">https://liuyueyi.github.io/hexblog</a></h3><p>一灰灰的个人博客，记录所有学习和工作中的博文，欢迎大家前去逛逛</p>
<h3 id="声明"><a href="#声明" class="headerlink" title="声明"></a>声明</h3><p>尽信书则不如，已上内容，纯属一家之言，因个人能力有限，难免有疏漏和错误之处，如发现bug或者有更好的建议，欢迎批评指正，不吝感激</p>
<ul>
<li>微博地址: <a href="https://weibo.com/p/1005052169825577/home" target="_blank" rel="noopener">小灰灰Blog</a></li>
<li>QQ： 一灰灰/3302797840</li>
</ul>
<h3 id="扫描关注"><a href="#扫描关注" class="headerlink" title="扫描关注"></a>扫描关注</h3><p><img src="https://raw.githubusercontent.com/liuyueyi/Source/master/img/info/blogInfoV2.png" alt="QrCode"></p>


            
        </div>
        
        <div class="level is-size-7 is-uppercase">
            <div class="level-start">
                <div class="level-item">
                    <span class="is-size-6 has-text-grey has-mr-7">#</span>
                    <a class="has-link-grey -link" href="/hexblog/tags/爬虫/">爬虫</a>
                </div>
            </div>
        </div>
        
        
        
        <div class="bdsharebuttonbox">
    <a href="#" class="bds_more" data-cmd="more"></a>
    <a href="#" class="bds_qzone" data-cmd="qzone" title="分享到QQ空间"></a>
    <a href="#" class="bds_tsina" data-cmd="tsina" title="分享到新浪微博"></a>
    <a href="#" class="bds_tqq" data-cmd="tqq" title="分享到腾讯微博"></a>
    <a href="#" class="bds_renren" data-cmd="renren" title="分享到人人网"></a>
    <a href="#" class="bds_weixin" data-cmd="weixin" title="分享到微信"></a>
</div>
<script>window._bd_share_config = { "common": { "bdSnsKey": {}, "bdText": "", "bdMini": "2", "bdPic": "", "bdStyle": "0", "bdSize": "16" }, "share": {} }; with (document) 0[(getElementsByTagName('head')[0] || body).appendChild(createElement('script')).src = 'http://bdimg.share.baidu.com/static/api/js/share.js?v=89860593.js?cdnversion=' + ~(-new Date() / 36e5)];</script>
        
    </div>
</div>



<div class="card">
    <div class="card-content">
        <h3 class="menu-label has-text-centered">喜欢这篇文章？打赏一下作者吧</h3>
        <div class="buttons is-centered">
            
                
<a class="button is-info donate">
    <span class="icon is-small">
        <i class="fab fa-alipay"></i>
    </span>
    <span>支付宝</span>
    <div class="qrcode"><img src="https://s3.mogucdn.com/mlcdn/c45406/180104_0e6afl33b23lacj6ji2d7d060aiak_798x800.png" alt="支付宝"></div>
</a>

                
                
<a class="button is-success donate">
    <span class="icon is-small">
        <i class="fab fa-weixin"></i>
    </span>
    <span>微信</span>
    <div class="qrcode"><img src="https://s11.mogucdn.com/mlcdn/c45406/180527_09cafb94a5g3lbd5ik6ke0hf649ff_800x798.jpg" alt="微信"></div>
</a>

                
        </div>
    </div>
</div>



<div class="card card-transparent">
    <div class="level post-navigation is-flex-wrap is-mobile">
        
        <div class="level-start">
            <a class="level level-item has-link-grey  article-nav-prev" href="/hexblog/2017/07/07/Java-动手写爬虫-三、爬取队列/">
                <i class="level-item fas fa-chevron-left"></i>
                <span class="level-item">Java 动手写爬虫: 三、爬取队列</span>
            </a>
        </div>
        
        
        <div class="level-end">
            <a class="level level-item has-link-grey  article-nav-next" href="/hexblog/2017/06/27/Java-动手写爬虫-一、实现一个最简单爬虫/">
                <span class="level-item">Java 动手写爬虫: 一、实现一个最简单爬虫</span>
                <i class="level-item fas fa-chevron-right"></i>
            </a>
        </div>
        
    </div>
</div>



<div class="card">
    <div class="card-content">
        <h3 class="title is-5 has-text-weight-normal">评论</h3>
        
<div id="comment-container"></div>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/gitalk@1.4.1/dist/gitalk.css">
<script src="https://cdn.jsdelivr.net/npm/gitalk@1.4.1/dist/gitalk.min.js"></script>
<script>
    var gitalk = new Gitalk({
        clientID: 'f4c5b85c7c1ceb8fbe2d',
        clientSecret: 'e8c365a05e6ab22272bee0b79ab27f69ff10a43a',
        id: 'b08b6e2aae259015ed216b5035ef6bea',
        repo: 'hexblog',
        owner: 'liuyueyi',
        admin: "liuyueyi"
    })
    gitalk.render('comment-container')
</script>

    </div>
</div>
</div>
                




<div class="column is-4-tablet is-4-desktop is-3-widescreen  has-order-1 column-left ">
    
        
<div class="card widget">
    <div class="card-content card-info">
        <nav class="level">
            <div class="level-item has-text-centered">
                <div>
                    
                        <img class="image is-128x128 has-mb-6" src="/hexblog/images/avatar.jpg" alt="一灰灰Blog">
                    
                    
                    <p class="is-size-4 is-block">
                        一灰灰Blog
                    </p>
                    
                    
                    <p class="is-size-6 is-block">
                        Java，服务器后端开发
                    </p>
                    
                    
                    <p class="is-size-6 is-flex is-flex-center has-text-grey">
                        <i class="fas fa-map-marker-alt has-mr-7"></i>
                        <span>Wuhan, China</span>
                    </p>
                    
                </div>
            </div>
        </nav>
        <nav class="level is-mobile">
            <div class="level-item has-text-centered is-marginless">
                <div>
                    
                    <p class="heading">
                        文章
                    </p>
                    <a href="/hexblog/archives">
                    <p class="title has-text-weight-normal">
                        269
                    </p>
                    </a>
                </div>
            </div>
            <div class="level-item has-text-centered is-marginless">
                <div>
                    
                    <p class="heading">
                        分类
                    </p>
                    <a href="/hexblog/categories">
                        <p class="title has-text-weight-normal">
                            70
                        </p>
                    </a>
                </div>
            </div>
            <div class="level-item has-text-centered is-marginless">
                <div>
                
                    <p class="heading">
                        标签
                    </p>
                    <a href="/hexblog/tags">
                    <p class="title has-text-weight-normal">
                        74
                    </p>
                    </a>
                </div>
            </div>
        </nav>
        <div class="level">
            <a class="level-item button is-link is-rounded" href="https://github.com/liuyueyi" target="_blank">
                关注我</a>
        </div>
        
        
        <div class="level is-mobile">
            
            <a class="level-item button is-white is-marginless" target="_blank"
                title="Github" href="https://github.com/liuyueyi">
                
                <i class="fab fa-github"></i>
                
            </a>
            
            <a class="level-item button is-white is-marginless" target="_blank"
                title="Gitee" href="https://gitee.com/liuyueyi">
                
                <i class="fab fa-gg"></i>
                
            </a>
            
            <a class="level-item button is-white is-marginless" target="_blank"
                title="Weibo" href="https://weibo.com/p/1005052169825577/home">
                
                <i class="fab fa-weibo"></i>
                
            </a>
            
            <a class="level-item button is-white is-marginless" target="_blank"
                title="weixin" href="https://s10.mogucdn.com/mlcdn/c45406/171229_1cgld3igbelkbc70cd8af1j3809kb_150x150.jpg">
                
                <i class="fab fa-weixin"></i>
                
            </a>
            
            <a class="level-item button is-white is-marginless" target="_blank"
                title="RSS" href="/hexblog/atom.xml">
                
                <i class="fas fa-rss"></i>
                
            </a>
            
        </div>
        
    </div>
</div>
    
        
    
        

<div class="card widget">
    <div class="card-content">
        <div class="menu">
        <h3 class="menu-label">
            链接
        </h3>
        <ul class="menu-list">
        
            <li>
                <a class="level is-mobile" href="http://spring.hhui.top" target="_blank">
                    <span class="level-left">
                        <span class="level-item">SpringBlog</span>
                    </span>
                    <span class="level-right">
                        <span class="level-item tag">spring.hhui.top</span>
                    </span>
                </a>
            </li>
        
            <li>
                <a class="level is-mobile" href="https://github.com/liuyueyi" target="_blank">
                    <span class="level-left">
                        <span class="level-item">GitHub</span>
                    </span>
                    <span class="level-right">
                        <span class="level-item tag">github.com</span>
                    </span>
                </a>
            </li>
        
            <li>
                <a class="level is-mobile" href="https://zweb.hhui.top" target="_blank">
                    <span class="level-left">
                        <span class="level-item">多媒体工具小站</span>
                    </span>
                    <span class="level-right">
                        <span class="level-item tag">zweb.hhui.top</span>
                    </span>
                </a>
            </li>
        
            <li>
                <a class="level is-mobile" href="https://mweb.hhui.top" target="_blank">
                    <span class="level-left">
                        <span class="level-item">每日十首古诗词</span>
                    </span>
                    <span class="level-right">
                        <span class="level-item tag">mweb.hhui.top</span>
                    </span>
                </a>
            </li>
        
        </ul>
        </div>
    </div>
</div>


    
        
<div class="card widget">
    <div class="card-content">
        <h3 class="menu-label">
            最新文章
        </h3>
        
        <article class="media">
            
            <div class="media-content">
                <div class="content">
                    <div><time class="has-text-grey is-size-7 is-uppercase" datetime="2020-03-28T11:27:12.000Z">2020-03-28</time></div>
                    <a href="/hexblog/2020/03/28/200328-MongoDb系列教程九-文档-Document-查询基础篇/" class="has-link-black-ter is-size-6">200328-MongoDb系列教程九：文档 Document 查询基础篇</a>
                    <p class="is-size-7 is-uppercase">
                        <a class="has-link-grey -link" href="/hexblog/categories/DB/">DB</a> / <a class="has-link-grey -link" href="/hexblog/categories/DB/Mongo/">Mongo</a>
                    </p>
                </div>
            </div>
        </article>
        
        <article class="media">
            
            <div class="media-content">
                <div class="content">
                    <div><time class="has-text-grey is-size-7 is-uppercase" datetime="2020-03-27T03:04:36.000Z">2020-03-27</time></div>
                    <a href="/hexblog/2020/03/27/200327-MongoDb系列教程八-文档-Document-更新姿势/" class="has-link-black-ter is-size-6">200327-MongoDb系列教程八：文档 Document 更新姿势</a>
                    <p class="is-size-7 is-uppercase">
                        <a class="has-link-grey -link" href="/hexblog/categories/DB/">DB</a> / <a class="has-link-grey -link" href="/hexblog/categories/DB/Mongo/">Mongo</a>
                    </p>
                </div>
            </div>
        </article>
        
        <article class="media">
            
            <div class="media-content">
                <div class="content">
                    <div><time class="has-text-grey is-size-7 is-uppercase" datetime="2020-03-26T10:02:03.000Z">2020-03-26</time></div>
                    <a href="/hexblog/2020/03/26/200326-MongoDb系列教程七-文档-Document-删除姿势/" class="has-link-black-ter is-size-6">200326-MongoDb系列教程七：文档 Document 删除姿势</a>
                    <p class="is-size-7 is-uppercase">
                        <a class="has-link-grey -link" href="/hexblog/categories/DB/">DB</a> / <a class="has-link-grey -link" href="/hexblog/categories/DB/Mongo/">Mongo</a>
                    </p>
                </div>
            </div>
        </article>
        
        <article class="media">
            
            <div class="media-content">
                <div class="content">
                    <div><time class="has-text-grey is-size-7 is-uppercase" datetime="2020-03-26T09:04:06.000Z">2020-03-26</time></div>
                    <a href="/hexblog/2020/03/26/200326-MongoDb系列教程六-文档-Document-插入姿势/" class="has-link-black-ter is-size-6">200326-MongoDb系列教程六：文档 Document 插入姿势</a>
                    <p class="is-size-7 is-uppercase">
                        <a class="has-link-grey -link" href="/hexblog/categories/DB/">DB</a> / <a class="has-link-grey -link" href="/hexblog/categories/DB/Mongo/">Mongo</a>
                    </p>
                </div>
            </div>
        </article>
        
        <article class="media">
            
            <div class="media-content">
                <div class="content">
                    <div><time class="has-text-grey is-size-7 is-uppercase" datetime="2020-03-26T08:52:20.000Z">2020-03-26</time></div>
                    <a href="/hexblog/2020/03/26/200326-MongoDb系列教程五-集合-Collection/" class="has-link-black-ter is-size-6">200326-MongoDb系列教程五：集合 Collection</a>
                    <p class="is-size-7 is-uppercase">
                        <a class="has-link-grey -link" href="/hexblog/categories/DB/">DB</a> / <a class="has-link-grey -link" href="/hexblog/categories/DB/Mongo/">Mongo</a>
                    </p>
                </div>
            </div>
        </article>
        
    </div>
</div>

    
        
<div class="card widget">
    <div class="card-content">
        <h3 class="menu-label">
            标签云
        </h3>
        <a href="/hexblog/tags/Android/" style="font-size: 10px;">Android</a> <a href="/hexblog/tags/AutoCloseable/" style="font-size: 10px;">AutoCloseable</a> <a href="/hexblog/tags/BloomFilter/" style="font-size: 10px;">BloomFilter</a> <a href="/hexblog/tags/BugFix/" style="font-size: 10.63px;">BugFix</a> <a href="/hexblog/tags/Bugfix/" style="font-size: 10.63px;">Bugfix</a> <a href="/hexblog/tags/Docker/" style="font-size: 11.88px;">Docker</a> <a href="/hexblog/tags/FastJson/" style="font-size: 10px;">FastJson</a> <a href="/hexblog/tags/Git/" style="font-size: 11.25px;">Git</a> <a href="/hexblog/tags/Groovy/" style="font-size: 10px;">Groovy</a> <a href="/hexblog/tags/Guava/" style="font-size: 11.25px;">Guava</a> <a href="/hexblog/tags/HashMap/" style="font-size: 10px;">HashMap</a> <a href="/hexblog/tags/IO/" style="font-size: 10.63px;">IO</a> <a href="/hexblog/tags/ImageMagic/" style="font-size: 10.63px;">ImageMagic</a> <a href="/hexblog/tags/InfluxDB/" style="font-size: 16.88px;">InfluxDB</a> <a href="/hexblog/tags/InputStream/" style="font-size: 10px;">InputStream</a> <a href="/hexblog/tags/JDK/" style="font-size: 15px;">JDK</a> <a href="/hexblog/tags/JVM/" style="font-size: 11.25px;">JVM</a> <a href="/hexblog/tags/Java/" style="font-size: 20px;">Java</a> <a href="/hexblog/tags/JavaAgent/" style="font-size: 10.63px;">JavaAgent</a> <a href="/hexblog/tags/JavaWeb/" style="font-size: 10.63px;">JavaWeb</a> <a href="/hexblog/tags/Jquery/" style="font-size: 10px;">Jquery</a> <a href="/hexblog/tags/Linux/" style="font-size: 15px;">Linux</a> <a href="/hexblog/tags/List/" style="font-size: 10px;">List</a> <a href="/hexblog/tags/MD5/" style="font-size: 10px;">MD5</a> <a href="/hexblog/tags/Map/" style="font-size: 10px;">Map</a> <a href="/hexblog/tags/Maven/" style="font-size: 10px;">Maven</a> <a href="/hexblog/tags/Mongo/" style="font-size: 10.63px;">Mongo</a> <a href="/hexblog/tags/MongoDB/" style="font-size: 10px;">MongoDB</a> <a href="/hexblog/tags/MongoDb/" style="font-size: 14.38px;">MongoDb</a> <a href="/hexblog/tags/MySql/" style="font-size: 10.63px;">MySql</a> <a href="/hexblog/tags/Mybatis/" style="font-size: 10.63px;">Mybatis</a> <a href="/hexblog/tags/Mysql/" style="font-size: 16.25px;">Mysql</a> <a href="/hexblog/tags/Nginx/" style="font-size: 11.25px;">Nginx</a> <a href="/hexblog/tags/OGNL/" style="font-size: 11.25px;">OGNL</a> <a href="/hexblog/tags/ProtoStuff/" style="font-size: 10.63px;">ProtoStuff</a> <a href="/hexblog/tags/Python/" style="font-size: 19.38px;">Python</a> <a href="/hexblog/tags/QuickAlarm/" style="font-size: 10px;">QuickAlarm</a> <a href="/hexblog/tags/RabbitMQ/" style="font-size: 13.75px;">RabbitMQ</a> <a href="/hexblog/tags/ReactJS/" style="font-size: 10px;">ReactJS</a> <a href="/hexblog/tags/Redis/" style="font-size: 14.38px;">Redis</a> <a href="/hexblog/tags/Shell/" style="font-size: 15.63px;">Shell</a> <a href="/hexblog/tags/Socket/" style="font-size: 10px;">Socket</a> <a href="/hexblog/tags/Solr/" style="font-size: 10px;">Solr</a> <a href="/hexblog/tags/Spring/" style="font-size: 18.75px;">Spring</a> <a href="/hexblog/tags/SpringBoot/" style="font-size: 10px;">SpringBoot</a> <a href="/hexblog/tags/Vue/" style="font-size: 10px;">Vue</a> <a href="/hexblog/tags/WebSocket/" style="font-size: 10px;">WebSocket</a> <a href="/hexblog/tags/Yaml/" style="font-size: 10px;">Yaml</a> <a href="/hexblog/tags/css/" style="font-size: 11.25px;">css</a> <a href="/hexblog/tags/ffmpeg/" style="font-size: 10px;">ffmpeg</a> <a href="/hexblog/tags/gitalk/" style="font-size: 10px;">gitalk</a> <a href="/hexblog/tags/hexo/" style="font-size: 10px;">hexo</a> <a href="/hexblog/tags/jdk/" style="font-size: 10px;">jdk</a> <a href="/hexblog/tags/logger/" style="font-size: 10px;">logger</a> <a href="/hexblog/tags/markdown/" style="font-size: 10px;">markdown</a> <a href="/hexblog/tags/python/" style="font-size: 10px;">python</a> <a href="/hexblog/tags/time/" style="font-size: 10px;">time</a> <a href="/hexblog/tags/乱码/" style="font-size: 10px;">乱码</a> <a href="/hexblog/tags/二维码/" style="font-size: 10px;">二维码</a> <a href="/hexblog/tags/分库分表/" style="font-size: 10px;">分库分表</a> <a href="/hexblog/tags/反射/" style="font-size: 11.25px;">反射</a> <a href="/hexblog/tags/工具/" style="font-size: 12.5px;">工具</a> <a href="/hexblog/tags/并发/" style="font-size: 11.25px;">并发</a> <a href="/hexblog/tags/序列化/" style="font-size: 10px;">序列化</a> <a href="/hexblog/tags/手记/" style="font-size: 13.13px;">手记</a> <a href="/hexblog/tags/技术方案/" style="font-size: 18.13px;">技术方案</a> <a href="/hexblog/tags/指南/" style="font-size: 13.13px;">指南</a> <a href="/hexblog/tags/教程/" style="font-size: 17.5px;">教程</a> <a href="/hexblog/tags/方案设计/" style="font-size: 10px;">方案设计</a> <a href="/hexblog/tags/时区/" style="font-size: 10px;">时区</a> <a href="/hexblog/tags/时间窗口/" style="font-size: 11.25px;">时间窗口</a> <a href="/hexblog/tags/爬虫/" style="font-size: 12.5px;">爬虫</a> <a href="/hexblog/tags/问题记录/" style="font-size: 10px;">问题记录</a> <a href="/hexblog/tags/随笔/" style="font-size: 10px;">随笔</a>
    </div>
</div>

    
        <div class="card widget">
    <div class="card-content">
        <div class="menu">
            <h3 class="menu-label">
                标签
            </h3>
            <div class="field is-grouped is-grouped-multiline">
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Android/">
                        <span class="tag">Android</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/AutoCloseable/">
                        <span class="tag">AutoCloseable</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/BloomFilter/">
                        <span class="tag">BloomFilter</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/BugFix/">
                        <span class="tag">BugFix</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Bugfix/">
                        <span class="tag">Bugfix</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Docker/">
                        <span class="tag">Docker</span>
                        <span class="tag is-grey">4</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/FastJson/">
                        <span class="tag">FastJson</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Git/">
                        <span class="tag">Git</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Groovy/">
                        <span class="tag">Groovy</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Guava/">
                        <span class="tag">Guava</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/HashMap/">
                        <span class="tag">HashMap</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/IO/">
                        <span class="tag">IO</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/ImageMagic/">
                        <span class="tag">ImageMagic</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/InfluxDB/">
                        <span class="tag">InfluxDB</span>
                        <span class="tag is-grey">16</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/InputStream/">
                        <span class="tag">InputStream</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/JDK/">
                        <span class="tag">JDK</span>
                        <span class="tag is-grey">11</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/JVM/">
                        <span class="tag">JVM</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Java/">
                        <span class="tag">Java</span>
                        <span class="tag is-grey">59</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/JavaAgent/">
                        <span class="tag">JavaAgent</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/JavaWeb/">
                        <span class="tag">JavaWeb</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Jquery/">
                        <span class="tag">Jquery</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Linux/">
                        <span class="tag">Linux</span>
                        <span class="tag is-grey">11</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/List/">
                        <span class="tag">List</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/MD5/">
                        <span class="tag">MD5</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Map/">
                        <span class="tag">Map</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Maven/">
                        <span class="tag">Maven</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Mongo/">
                        <span class="tag">Mongo</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/MongoDB/">
                        <span class="tag">MongoDB</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/MongoDb/">
                        <span class="tag">MongoDb</span>
                        <span class="tag is-grey">9</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/MySql/">
                        <span class="tag">MySql</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Mybatis/">
                        <span class="tag">Mybatis</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Mysql/">
                        <span class="tag">Mysql</span>
                        <span class="tag is-grey">15</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Nginx/">
                        <span class="tag">Nginx</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/OGNL/">
                        <span class="tag">OGNL</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/ProtoStuff/">
                        <span class="tag">ProtoStuff</span>
                        <span class="tag is-grey">2</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Python/">
                        <span class="tag">Python</span>
                        <span class="tag is-grey">32</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/QuickAlarm/">
                        <span class="tag">QuickAlarm</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/RabbitMQ/">
                        <span class="tag">RabbitMQ</span>
                        <span class="tag is-grey">8</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/ReactJS/">
                        <span class="tag">ReactJS</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Redis/">
                        <span class="tag">Redis</span>
                        <span class="tag is-grey">9</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Shell/">
                        <span class="tag">Shell</span>
                        <span class="tag is-grey">13</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Socket/">
                        <span class="tag">Socket</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Solr/">
                        <span class="tag">Solr</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Spring/">
                        <span class="tag">Spring</span>
                        <span class="tag is-grey">25</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/SpringBoot/">
                        <span class="tag">SpringBoot</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Vue/">
                        <span class="tag">Vue</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/WebSocket/">
                        <span class="tag">WebSocket</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/Yaml/">
                        <span class="tag">Yaml</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/css/">
                        <span class="tag">css</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/ffmpeg/">
                        <span class="tag">ffmpeg</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/gitalk/">
                        <span class="tag">gitalk</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/hexo/">
                        <span class="tag">hexo</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/jdk/">
                        <span class="tag">jdk</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/logger/">
                        <span class="tag">logger</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/markdown/">
                        <span class="tag">markdown</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/python/">
                        <span class="tag">python</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/time/">
                        <span class="tag">time</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/乱码/">
                        <span class="tag">乱码</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/二维码/">
                        <span class="tag">二维码</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/分库分表/">
                        <span class="tag">分库分表</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/反射/">
                        <span class="tag">反射</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/工具/">
                        <span class="tag">工具</span>
                        <span class="tag is-grey">5</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/并发/">
                        <span class="tag">并发</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/序列化/">
                        <span class="tag">序列化</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/手记/">
                        <span class="tag">手记</span>
                        <span class="tag is-grey">6</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/技术方案/">
                        <span class="tag">技术方案</span>
                        <span class="tag is-grey">22</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/指南/">
                        <span class="tag">指南</span>
                        <span class="tag is-grey">6</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/教程/">
                        <span class="tag">教程</span>
                        <span class="tag is-grey">20</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/方案设计/">
                        <span class="tag">方案设计</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/时区/">
                        <span class="tag">时区</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/时间窗口/">
                        <span class="tag">时间窗口</span>
                        <span class="tag is-grey">3</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/爬虫/">
                        <span class="tag">爬虫</span>
                        <span class="tag is-grey">5</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/问题记录/">
                        <span class="tag">问题记录</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
                <div class="control">
                    <a class="tags has-addons" href="/hexblog/tags/随笔/">
                        <span class="tag">随笔</span>
                        <span class="tag is-grey">1</span>
                    </a>
                </div>
                
            </div>
        </div>
    </div>
</div>
    
    
        <div class="column-right-shadow is-hidden-widescreen ">
        
            
<div class="card widget">
    <div class="card-content">
        <div class="menu">
            <h3 class="menu-label">
                分类
            </h3>
            <ul class="menu-list">
            <li>
        <a class="level is-marginless" href="/hexblog/categories/DB/">
            <span class="level-start">
                <span class="level-item">DB</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">41</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/InfluxDB/">
            <span class="level-start">
                <span class="level-item">InfluxDB</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">16</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/Mongo/">
            <span class="level-start">
                <span class="level-item">Mongo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">11</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/Mysql/">
            <span class="level-start">
                <span class="level-item">Mysql</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">13</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/分库分表/">
            <span class="level-start">
                <span class="level-item">分库分表</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/">
            <span class="level-start">
                <span class="level-item">Java</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">52</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Agent/">
            <span class="level-start">
                <span class="level-item">Agent</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Android/">
            <span class="level-start">
                <span class="level-item">Android</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Bugfix/">
            <span class="level-start">
                <span class="level-item">Bugfix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/IO/">
            <span class="level-start">
                <span class="level-item">IO</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JDK/">
            <span class="level-start">
                <span class="level-item">JDK</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">22</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JVM/">
            <span class="level-start">
                <span class="level-item">JVM</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JavaWeb/">
            <span class="level-start">
                <span class="level-item">JavaWeb</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/其他/">
            <span class="level-start">
                <span class="level-item">其他</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/问题记录/">
            <span class="level-start">
                <span class="level-item">问题记录</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/">
            <span class="level-start">
                <span class="level-item">Python</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">33</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/Mongo/">
            <span class="level-start">
                <span class="level-item">Mongo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/MySql/">
            <span class="level-start">
                <span class="level-item">MySql</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/教程/">
            <span class="level-start">
                <span class="level-item">教程</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">25</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/采坑记录/">
            <span class="level-start">
                <span class="level-item">采坑记录</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/项目/">
            <span class="level-start">
                <span class="level-item">项目</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/">
            <span class="level-start">
                <span class="level-item">Quick系列</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">30</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickAlarm/">
            <span class="level-start">
                <span class="level-item">QuickAlarm</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickCrawler/">
            <span class="level-start">
                <span class="level-item">QuickCrawler</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">5</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickFix/">
            <span class="level-start">
                <span class="level-item">QuickFix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickMedia/">
            <span class="level-start">
                <span class="level-item">QuickMedia</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickSpi/">
            <span class="level-start">
                <span class="level-item">QuickSpi</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickTask/">
            <span class="level-start">
                <span class="level-item">QuickTask</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">5</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/">
            <span class="level-start">
                <span class="level-item">Shell</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">39</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/CMD/">
            <span class="level-start">
                <span class="level-item">CMD</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">18</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Docker/">
            <span class="level-start">
                <span class="level-item">Docker</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Git/">
            <span class="level-start">
                <span class="level-item">Git</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Maven/">
            <span class="level-start">
                <span class="level-item">Maven</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Ngins/">
            <span class="level-start">
                <span class="level-item">Ngins</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Nginx/">
            <span class="level-start">
                <span class="level-item">Nginx</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/环境搭建/">
            <span class="level-start">
                <span class="level-item">环境搭建</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/">
            <span class="level-start">
                <span class="level-item">前端</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Chrome/">
            <span class="level-start">
                <span class="level-item">Chrome</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Css/">
            <span class="level-start">
                <span class="level-item">Css</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Jquery/">
            <span class="level-start">
                <span class="level-item">Jquery</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/ReactJS/">
            <span class="level-start">
                <span class="level-item">ReactJS</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Vue/">
            <span class="level-start">
                <span class="level-item">Vue</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/">
            <span class="level-start">
                <span class="level-item">工具</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/工具类/">
            <span class="level-start">
                <span class="level-item">工具类</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/插件系列/">
            <span class="level-start">
                <span class="level-item">插件系列</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/">
            <span class="level-start">
                <span class="level-item">开源</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">46</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Guava/">
            <span class="level-start">
                <span class="level-item">Guava</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Hexo/">
            <span class="level-start">
                <span class="level-item">Hexo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Hystrix/">
            <span class="level-start">
                <span class="level-item">Hystrix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Mybatis/">
            <span class="level-start">
                <span class="level-item">Mybatis</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/OGNL/">
            <span class="level-start">
                <span class="level-item">OGNL</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/RabbitMQ/">
            <span class="level-start">
                <span class="level-item">RabbitMQ</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Redis/">
            <span class="level-start">
                <span class="level-item">Redis</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Solr/">
            <span class="level-start">
                <span class="level-item">Solr</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Spring/">
            <span class="level-start">
                <span class="level-item">Spring</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">24</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Yaml/">
            <span class="level-start">
                <span class="level-item">Yaml</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/火花/">
            <span class="level-start">
                <span class="level-item">火花</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/">
            <span class="level-start">
                <span class="level-item">随笔</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">7</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/idea/">
            <span class="level-start">
                <span class="level-item">idea</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/吐槽/">
            <span class="level-start">
                <span class="level-item">吐槽</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li>
            </ul>
        </div>
    </div>
</div>
        
            <div class="card widget">
    <div class="card-content">
        <div class="menu">
        <h3 class="menu-label">
            归档
        </h3>
        <ul class="menu-list">
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/03/">
                <span class="level-start">
                    <span class="level-item">三月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">14</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/02/">
                <span class="level-start">
                    <span class="level-item">二月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/01/">
                <span class="level-start">
                    <span class="level-item">一月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">5</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/11/">
                <span class="level-start">
                    <span class="level-item">十一月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">6</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/10/">
                <span class="level-start">
                    <span class="level-item">十月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/09/">
                <span class="level-start">
                    <span class="level-item">九月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/08/">
                <span class="level-start">
                    <span class="level-item">八月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">5</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/07/">
                <span class="level-start">
                    <span class="level-item">七月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/06/">
                <span class="level-start">
                    <span class="level-item">六月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/05/">
                <span class="level-start">
                    <span class="level-item">五月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">12</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/04/">
                <span class="level-start">
                    <span class="level-item">四月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/03/">
                <span class="level-start">
                    <span class="level-item">三月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/02/">
                <span class="level-start">
                    <span class="level-item">二月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">3</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/01/">
                <span class="level-start">
                    <span class="level-item">一月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">14</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">8</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/11/">
                <span class="level-start">
                    <span class="level-item">十一月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">10</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/10/">
                <span class="level-start">
                    <span class="level-item">十月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/09/">
                <span class="level-start">
                    <span class="level-item">九月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">13</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/08/">
                <span class="level-start">
                    <span class="level-item">八月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/07/">
                <span class="level-start">
                    <span class="level-item">七月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">23</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/06/">
                <span class="level-start">
                    <span class="level-item">六月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">22</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/05/">
                <span class="level-start">
                    <span class="level-item">五月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/04/">
                <span class="level-start">
                    <span class="level-item">四月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/03/">
                <span class="level-start">
                    <span class="level-item">三月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">16</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/02/">
                <span class="level-start">
                    <span class="level-item">二月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">10</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/01/">
                <span class="level-start">
                    <span class="level-item">一月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">13</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">6</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/09/">
                <span class="level-start">
                    <span class="level-item">九月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/08/">
                <span class="level-start">
                    <span class="level-item">八月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">3</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/07/">
                <span class="level-start">
                    <span class="level-item">七月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/06/">
                <span class="level-start">
                    <span class="level-item">六月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/05/">
                <span class="level-start">
                    <span class="level-item">五月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        </ul>
        </div>
    </div>
</div>
        
        </div>
    
</div>

                




<div class="column is-4-tablet is-4-desktop is-2-widescreen is-hidden-touch is-hidden-desktop-only has-order-3 column-right ">
    
        
<div class="card widget">
    <div class="card-content">
        <div class="menu">
            <h3 class="menu-label">
                分类
            </h3>
            <ul class="menu-list">
            <li>
        <a class="level is-marginless" href="/hexblog/categories/DB/">
            <span class="level-start">
                <span class="level-item">DB</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">41</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/InfluxDB/">
            <span class="level-start">
                <span class="level-item">InfluxDB</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">16</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/Mongo/">
            <span class="level-start">
                <span class="level-item">Mongo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">11</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/Mysql/">
            <span class="level-start">
                <span class="level-item">Mysql</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">13</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/DB/分库分表/">
            <span class="level-start">
                <span class="level-item">分库分表</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/">
            <span class="level-start">
                <span class="level-item">Java</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">52</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Agent/">
            <span class="level-start">
                <span class="level-item">Agent</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Android/">
            <span class="level-start">
                <span class="level-item">Android</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/Bugfix/">
            <span class="level-start">
                <span class="level-item">Bugfix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/IO/">
            <span class="level-start">
                <span class="level-item">IO</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JDK/">
            <span class="level-start">
                <span class="level-item">JDK</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">22</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JVM/">
            <span class="level-start">
                <span class="level-item">JVM</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/JavaWeb/">
            <span class="level-start">
                <span class="level-item">JavaWeb</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/其他/">
            <span class="level-start">
                <span class="level-item">其他</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Java/问题记录/">
            <span class="level-start">
                <span class="level-item">问题记录</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/">
            <span class="level-start">
                <span class="level-item">Python</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">33</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/Mongo/">
            <span class="level-start">
                <span class="level-item">Mongo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/MySql/">
            <span class="level-start">
                <span class="level-item">MySql</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/教程/">
            <span class="level-start">
                <span class="level-item">教程</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">25</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/采坑记录/">
            <span class="level-start">
                <span class="level-item">采坑记录</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Python/项目/">
            <span class="level-start">
                <span class="level-item">项目</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/">
            <span class="level-start">
                <span class="level-item">Quick系列</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">30</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickAlarm/">
            <span class="level-start">
                <span class="level-item">QuickAlarm</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickCrawler/">
            <span class="level-start">
                <span class="level-item">QuickCrawler</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">5</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickFix/">
            <span class="level-start">
                <span class="level-item">QuickFix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickMedia/">
            <span class="level-start">
                <span class="level-item">QuickMedia</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickSpi/">
            <span class="level-start">
                <span class="level-item">QuickSpi</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Quick系列/QuickTask/">
            <span class="level-start">
                <span class="level-item">QuickTask</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">5</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/">
            <span class="level-start">
                <span class="level-item">Shell</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">39</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/CMD/">
            <span class="level-start">
                <span class="level-item">CMD</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">18</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Docker/">
            <span class="level-start">
                <span class="level-item">Docker</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Git/">
            <span class="level-start">
                <span class="level-item">Git</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Maven/">
            <span class="level-start">
                <span class="level-item">Maven</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Ngins/">
            <span class="level-start">
                <span class="level-item">Ngins</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/Nginx/">
            <span class="level-start">
                <span class="level-item">Nginx</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/Shell/环境搭建/">
            <span class="level-start">
                <span class="level-item">环境搭建</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/">
            <span class="level-start">
                <span class="level-item">前端</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Chrome/">
            <span class="level-start">
                <span class="level-item">Chrome</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Css/">
            <span class="level-start">
                <span class="level-item">Css</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Jquery/">
            <span class="level-start">
                <span class="level-item">Jquery</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/ReactJS/">
            <span class="level-start">
                <span class="level-item">ReactJS</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/前端/Vue/">
            <span class="level-start">
                <span class="level-item">Vue</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/">
            <span class="level-start">
                <span class="level-item">工具</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">9</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/工具类/">
            <span class="level-start">
                <span class="level-item">工具类</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">6</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/工具/插件系列/">
            <span class="level-start">
                <span class="level-item">插件系列</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/">
            <span class="level-start">
                <span class="level-item">开源</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">46</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Guava/">
            <span class="level-start">
                <span class="level-item">Guava</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Hexo/">
            <span class="level-start">
                <span class="level-item">Hexo</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Hystrix/">
            <span class="level-start">
                <span class="level-item">Hystrix</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Mybatis/">
            <span class="level-start">
                <span class="level-item">Mybatis</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">2</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/OGNL/">
            <span class="level-start">
                <span class="level-item">OGNL</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/RabbitMQ/">
            <span class="level-start">
                <span class="level-item">RabbitMQ</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">8</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Redis/">
            <span class="level-start">
                <span class="level-item">Redis</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">3</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Solr/">
            <span class="level-start">
                <span class="level-item">Solr</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Spring/">
            <span class="level-start">
                <span class="level-item">Spring</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">24</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/开源/Yaml/">
            <span class="level-start">
                <span class="level-item">Yaml</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li><li>
        <a class="level is-marginless" href="/hexblog/categories/火花/">
            <span class="level-start">
                <span class="level-item">火花</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">4</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/">
            <span class="level-start">
                <span class="level-item">随笔</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">7</span>
            </span>
        </a><ul><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/idea/">
            <span class="level-start">
                <span class="level-item">idea</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li><li>
        <a class="level is-marginless" href="/hexblog/categories/随笔/吐槽/">
            <span class="level-start">
                <span class="level-item">吐槽</span>
            </span>
            <span class="level-end">
                <span class="level-item tag">1</span>
            </span>
        </a></li></ul></li>
            </ul>
        </div>
    </div>
</div>
    
        <div class="card widget">
    <div class="card-content">
        <div class="menu">
        <h3 class="menu-label">
            归档
        </h3>
        <ul class="menu-list">
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/03/">
                <span class="level-start">
                    <span class="level-item">三月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">14</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/02/">
                <span class="level-start">
                    <span class="level-item">二月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2020/01/">
                <span class="level-start">
                    <span class="level-item">一月 2020</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">5</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/11/">
                <span class="level-start">
                    <span class="level-item">十一月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">6</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/10/">
                <span class="level-start">
                    <span class="level-item">十月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/09/">
                <span class="level-start">
                    <span class="level-item">九月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/08/">
                <span class="level-start">
                    <span class="level-item">八月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">5</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/07/">
                <span class="level-start">
                    <span class="level-item">七月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/06/">
                <span class="level-start">
                    <span class="level-item">六月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/05/">
                <span class="level-start">
                    <span class="level-item">五月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">12</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/04/">
                <span class="level-start">
                    <span class="level-item">四月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/03/">
                <span class="level-start">
                    <span class="level-item">三月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">7</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/02/">
                <span class="level-start">
                    <span class="level-item">二月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">3</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2019/01/">
                <span class="level-start">
                    <span class="level-item">一月 2019</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">14</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">8</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/11/">
                <span class="level-start">
                    <span class="level-item">十一月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">10</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/10/">
                <span class="level-start">
                    <span class="level-item">十月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/09/">
                <span class="level-start">
                    <span class="level-item">九月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">13</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/08/">
                <span class="level-start">
                    <span class="level-item">八月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/07/">
                <span class="level-start">
                    <span class="level-item">七月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">23</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/06/">
                <span class="level-start">
                    <span class="level-item">六月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">22</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/05/">
                <span class="level-start">
                    <span class="level-item">五月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/04/">
                <span class="level-start">
                    <span class="level-item">四月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">11</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/03/">
                <span class="level-start">
                    <span class="level-item">三月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">16</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/02/">
                <span class="level-start">
                    <span class="level-item">二月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">10</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2018/01/">
                <span class="level-start">
                    <span class="level-item">一月 2018</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">13</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/12/">
                <span class="level-start">
                    <span class="level-item">十二月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">6</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/09/">
                <span class="level-start">
                    <span class="level-item">九月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">1</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/08/">
                <span class="level-start">
                    <span class="level-item">八月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">3</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/07/">
                <span class="level-start">
                    <span class="level-item">七月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/06/">
                <span class="level-start">
                    <span class="level-item">六月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">2</span>
                </span>
            </a>
        </li>
        
        <li>
            <a class="level is-marginless" href="/hexblog/archives/2017/05/">
                <span class="level-start">
                    <span class="level-item">五月 2017</span>
                </span>
                <span class="level-end">
                    <span class="level-item tag">4</span>
                </span>
            </a>
        </li>
        
        </ul>
        </div>
    </div>
</div>
    
    
</div>

            </div>
        </div>
    </section>
    <footer class="footer">
    <div class="container">
        <div class="level">
            <div class="level-start has-text-centered-mobile">
                <a class="footer-logo is-block has-mb-6" href="/hexblog/">
                
                    <img src="/hexblog/images/avatar.jpg" alt="Java 动手写爬虫: 二、 深度爬取" height="28">
                
                </a>
                <p class="is-size-7">
                &copy; 2020 YiHui&nbsp;
                Powered by <a href="https://hexo.io/" target="_blank">Hexo</a> & <a
                        href="https://github.com/ppoffice/hexo-theme-icarus" target="_blank">Icarus</a>
                
                </p>
            </div>
            <div class="level-end">
            
                <div class="field has-addons is-flex-center-mobile has-mt-5-mobile is-flex-wrap is-flex-middle">
                
                
                <p class="control">
                    <a class="button is-white is-large" target="_blank" title="Download on GitHub" href="https://github.com/liuyueyi">
                        
                        <i class="fab fa-github"></i>
                        
                    </a>
                </p>
                
                <p class="control">
                    <a class="button is-white is-large" target="_blank" title="Chart in Weibo" href="https://weibo.com/p/1005052169825577/home">
                        
                        <i class="fab fa-weibo"></i>
                        
                    </a>
                </p>
                
                <p class="control">
                    <a class="button is-white is-large" target="_blank" title="Frends with me" href="https://s10.mogucdn.com/mlcdn/c45406/171229_1cgld3igbelkbc70cd8af1j3809kb_150x150.jpg">
                        
                        <i class="fab fa-weixin"></i>
                        
                    </a>
                </p>
                
                </div>
            
            </div>
        </div>
    </div>
    <hr/>
    <div id="foot-pannel">
        <div class="outer">
            <div class="foot-column">
                <!--关注我们-->
                <div class="widget-wrap widget-list">
                    <h3 class="widget-title">更多平台</h3>
                    <div class="widget">
                        <ul>
                            <li>
                                <a href="//blog.hhui.top/">一灰灰Blog</a>
                            </li>
                        
                            <li>
                                <a href="//juejin.im/user/5a2a4b095188252ae93adbbf/posts">掘金</a>
                            </li>
                        
                            <li>
                                <a href="//my.oschina.net/u/566591">开源中国</a>
                            </li>
                        
                            <li>
                                <a href="//blog.csdn.net/liuyueyi25">CSDN</a>
                            </li>
                        
                            <li>
                                <a href="//www.jianshu.com/u/5902ab08e670">简书</a>
                            </li>
                        
                            <li>
                                <a href="//cloud.tencent.com/developer/column/1847">云+</a>
                            </li>
                        
                            <li>
                                <a href="//www.toutiao.com/c/user/69862071663/#mid=1579653107239950">头条</a>
                            </li>
                        
                            <li>
                                <a href="//github.com/liuyueyi">GitHub</a>
                            </li>
                        
                            <li>
                                <a href="//gitee.com/liuyueyi">Gitee</a>
                            </li>
                            
                        </ul>
                    </div>
                </div>
            </div>
            <div class="foot-column">
                <!--联系合作-->
                <div class="widget-wrap widget-list">
                    <h3 class="widget-title">一灰灰Blog</h3>
                    <div class="widget">
                        <ul>
                            
                                <li>
                                    <a href="#">QQ : 3302797840</a>
                                </li>
                            
                                <li>
                                    <a href="#">微信 : liuyueyi25</a>
                                </li>
                            
                                <li>
                                    <a href="#">邮箱 : bangzewu@126.com</a>
                                </li>
                            
                                <li>
                                    <a href="#">微博 : 一灰灰blog</a>
                                </li>
                            
                        </ul>
                    </div>
                </div>
                <div class="widget-wrap widget-list">
                    <h3 class="widget-title">友情链接</h3>
                    <div class="widget">
                        <ul>
                            
                                <span class="label">
                                    <a target="_blank" href="//zweb.hhui.top">zweb多媒体工具页</a>
                                </span>
                            
                                <span class="label">
                                    <a target="_blank" href="//mweb.hhui.top">mweb古诗选</a>
                                </span>
                            
                        </ul>
                    </div>
                </div>
            </div>
            <div class="foot-column">
                <!--友情链接-->
                <div class="widget-wrap widget-list">
                    <h3 class="widget-title">知识星球</h3>
                    <div class="widget">
                        <img style="width: 200px;" src="/hexblog/imgs/info/xingqiu.png">
                    </div>
                </div>
                
            </div>
            <div class="foot-column">
                <div class="widget-wrap widget-list">
                    <h3 class="widget-title">公众号</h3>
                    <div class="widget">
                        <img style="width: 200px;" src="/hexblog/imgs/info/wx.jpg">
                    </div>
                </div>
            </div>
        </div>
    </div>
    <div style="padding-top:20em">
        <hr/>
        <div id="foot-info">
            <span style="margin:4px;font-size:1em">
                &copy; 2020 <a target='_blank' href='https://github.com/liuyueyi'>一灰灰Blog</a> 版权所有 | <a href="http://www.beian.miit.gov.cn" target="_blank">鄂ICP备18017282号</a>
            </span>
            <br/>
            <span style="margin:4px;font-size:1em">
                <script type="text/javascript">document.write(unescape("%3Cspan id='cnzz_stat_icon_1278691600'%3E%3C/span%3E%3Cscript src='https://s9.cnzz.com/z_stat.php%3Fid%3D1278691600%26online%3D1%26show%3Dline' type='text/javascript'%3E%3C/script%3E"));</script>
            </span>
            <span style="margin:1em;font-size:1.4em">
                <label id="self_count_cnt"><br>本站总访量: <span class="visit_cnt">69330</span> | 总访问人次: <span class="visit_cnt">11586</span> | 恭喜您为第 <span class="visit_cnt">10840</span> 访问者</label>
            </span>
            <script src="/hexblog/js/count.js"></script>
        </div>
    </div>
</footer>
    <script src="https://cdn.jsdelivr.net/npm/jquery@3.3.1/dist/jquery.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/moment@2.22.2/min/moment-with-locales.min.js"></script>
<script>moment.locale("zh-CN");</script>


    
    
    
    <script src="/hexblog/js/animation.js"></script>
    

    
    
    
    <script src="https://cdn.jsdelivr.net/npm/lightgallery@1.6.8/dist/js/lightgallery.min.js" defer></script>
    <script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js" defer></script>
    <script src="/hexblog/js/gallery.js" defer></script>
    

    
    

<div id="outdated">
    <h6>Your browser is out-of-date!</h6>
    <p>Update your browser to view this website correctly. <a id="btnUpdateBrowser" href="http://outdatedbrowser.com/">Update
            my browser now </a></p>
    <p class="last"><a href="#" id="btnCloseUpdateBrowser" title="Close">&times;</a></p>
</div>
<script src="https://cdn.jsdelivr.net/npm/outdatedbrowser@1.1.5/outdatedbrowser/outdatedbrowser.min.js" defer></script>
<script>
    document.addEventListener("DOMContentLoaded", function () {
        outdatedBrowser({
            bgColor: '#f25648',
            color: '#ffffff',
            lowerThan: 'flex'
        });
    });
</script>


    
    
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.5/unpacked/MathJax.js?config=TeX-MML-AM_CHTML" defer></script>
<script>
document.addEventListener('DOMContentLoaded', function () {
    MathJax.Hub.Config({
        'HTML-CSS': {
            matchFontHeight: false
        },
        SVG: {
            matchFontHeight: false
        },
        CommonHTML: {
            matchFontHeight: false
        },
        tex2jax: {
            inlineMath: [
                ['$','$'],
                ['\\(','\\)']
            ]
        }
    });
});
</script>

    
    

<a id="back-to-top" title="回到顶端" href="javascript:;">
    <i class="fas fa-chevron-up"></i>
</a>
<script src="/hexblog/js/back-to-top.js" defer></script>


    
    

    
    
    
    

    
    
    
    
    
    <script src="https://cdn.jsdelivr.net/npm/clipboard@2.0.4/dist/clipboard.min.js" defer></script>
    <script src="/hexblog/js/clipboard.js" defer></script>
    

    
    
    


<script src="/hexblog/js/main.js" defer></script>

    
    <div class="searchbox ins-search">
    <div class="searchbox-container ins-search-container">
        <div class="searchbox-input-wrapper">
            <input type="text" class="searchbox-input ins-search-input" placeholder="想要查找什么..." />
            <span class="searchbox-close ins-close ins-selectable"><i class="fa fa-times-circle"></i></span>
        </div>
        <div class="searchbox-result-wrapper ins-section-wrapper">
            <div class="ins-section-container"></div>
        </div>
    </div>
</div>
<script>
    (function (window) {
        var INSIGHT_CONFIG = {
            TRANSLATION: {
                POSTS: '文章',
                PAGES: '页面',
                CATEGORIES: '分类',
                TAGS: '标签',
                UNTITLED: '(无标题)',
            },
            CONTENT_URL: '/hexblog/content.json',
        };
        window.INSIGHT_CONFIG = INSIGHT_CONFIG;
    })(window);
</script>
<script src="/hexblog/js/insight.js" defer></script>
<link rel="stylesheet" href="/hexblog/css/search.css">
<link rel="stylesheet" href="/hexblog/css/insight.css">
    
</body>
</html>